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ABSTRACT 
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interpretation of qualitative data. It explores a grounded theory 
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construction of categories. It focuses on ways of organizing data and 
attaching meaning, as research problems embedded in cultural context 
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was used. We wanted to know how different ways of categorizing data 
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QUALITATIVE ANALYSIS OF A SYNTHETIC CULTURE: 
A METHODOLOGICAL INQUIRY 

Suzanne MacDonald 
Isadora Newman 
Kyle Waite 
Brad Potts 

University of Akron 

ABSTRACT 

The study is a methodological inquiry into the interpretation of qualitative data. It explores a 
grounded theory approach to the synthesis of data, and examines, in particular, construction of 
categories. It focuses on ways of organizing data and attaching meaning, as research problems 
embedded in cultural context are explored. A qualitative research training task, with 4-7 subjects 
per group (4 or 5 groups per class) evaluating comic strip cuiture.was used. We wanted to know 
how different ways of categorizing data lead to different interpretations of comic strip culture. In 
this regard we were looking at a) consistency within groups, and b) how groups differ. 
Implications of this study center around the idiosyncratic nature of qualitative research, issues 
related to generalizability. and relationships between training and non-training of researchers to 
the interpretation of data. 

Perspective 

Qualitative research has gained in frequency of use but with 
proportionately fewer people well-trained in the methodology applying to its 
use. With this increase in usage there is concomitantly an increase in the need 
to understand some of the concerns innate in the procedures used when 
conducting qualitative investigations. It is recognized by almost all authorities 
(Eisner & Peshkin, 1990; Guba & Lincoln, 1982; LeCompte & Goetz, 1982) that 
qualitative research has a high subjective component in its interpretation and 
that this has been one of the- major concerns and criticisms levied at this 
particular approach. They recognize that qualitative research has been 
criticized for its subjectivity. The rebuttal to such criticism tends to rest on the 
assumption that training in coding methodology as well as other aspects of 
qualitative procedures will decrease concerns associated with some of the 
major criticisms. Based upon the above argument, the investigators designed a 
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methodological inquiry examining the potential effect of coding training on the 
interpretation of a cartoon culture. 

As a methodological inquiry into the interpretation of qualitative data, the 
study explores a grounded theory approach to the synthesis of data : examining, 
in particular, the i dentification and building of categories, as one part of the 
coding, categorizing, and thematic development sequence. It focuses on ways 
of organizing data and attaching meaning, as research problems embedded in 
cultural context, are explored as part of the continuing debate concerning 
qualitative research (Constas, 1 992). 

The research addresses some key components of an ongoing debate 
over the value and place of qualitative approaches to the understanding of 
problems in education and related fields. Eisner and Peshkin 0990) stress the 
need to explore and further develop models of qualitative research acceptable 
to the research community, and to which "educational researchers with a 
qualitataive bent could turn for direction." Newman and Benz (1992) ask that 
educational researchers reject the qualitative-quantitative dichotomy and think 
in terms of a synthesis of the two approaches. This perspective involves, 
among other things, renewed examination of both research approaches in 
terms of a broader paradigm or frame of reference than has often been used in 
the past. 

This study focuses on the data analysis phase of ethnographic research. 
The importance of the study is related to the tremendous complexity and 
layering of ethnographic interpretation as described by Geertz, when he 
observed ". . . that what we call our data are really our own constructions of 
other people's constructions of what they and their compatriots are up to-is 
obscured because most of what we need to comprehend a particular event, 
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ritual, custom, idea, or whatever is insinuated as backgound information before 
the thing itself is directly examined (1973, p. 9)." 

This particular research focuses on qualitative inquiry in relation to 
issues of reliability, validity, and generalization-concerns often raised by 
educational researchers, especially by "quantitative" researchers. Addressing 
its inappropriateness for generalization, the value of qualitative research is 
often described as in the depth and richness of description it provides in 
particular and idiosyncratic cases, and npl in generalization to other cases 
(Erickson, 1988; Peshkin, 1993). However, there are some recent claims by 
oualitative researchers such as Polkinghorne (1991) that qualitative research 
can be generalized beyond the specific case. 
Objectives 

Objective: To investigate some methodological aspects of qualitative 
analysis, i.e., 

1 ) To replicate findings from a previous pilot study 

2) To estimate the effect of training students in coding techniques 
on change in their interpretation 

3) To estimate the effect of training students in coding techniques 

towards increasing group consensus (within-group change) 

4) To estimate the effect of training students in coding techniques 

towards increasing generalizability across groups (across- 
group change) 
Method and Data Source 

A comic strip culture was used as a common data set for studying the 
elicitation of categories in qualitative analysis by 1) individuals given the same 
data, and 2) groups consisting of the individuals who had already processed 
the data individually. Comparisons were made of 1 ) individual responses , 
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2) intra-group responses, and 3) inter-group responses. A qualitative research 
training task, with 4-7 member groups evaluating comic strip culture, was used. 
We wanted to know how different ways of categorizing data lead to different 
interpretations of comic strip culture. In this regard we were looking at 
a) consistency within groups, and b) how groups differ. 

We started with four graduate students who served as interviewees after 
reading comic strip material to be used with a graduate class later. Questions 
and approach to the cartoon culture simulation were modified on the basis of 
what we learned. In Week 9 of the semester, twenty-seven students in a 
graduate course focusing on education in cultural context, participated in a 
qualitative inquiry into the nature of culture and its analysis through a simulation 
using ethnographic techniques. In this initial inquiry (Study 1), students doing 
the analysis had not been trained in ethnographic techniques, although they 
had some general exposure to the topic. They had, however, been exposed 
over several sessions to characteristics of culture, and a range of concepts 
associated with culture and sub-cultures. 

The class activity was titled MAKING MEANING: AN EXPLORATION OF 
COMIC STRIP CULTURE , and resulted in 27 individual and 4 group responses 
to the following set of questions: 1) Who are the main characters (in this 
culture)? 2) Which characters have the most prestige (in this culture)? Why do 
you think this? *3) Describe this culture. What are the cultural values? 4) What 
general reaction and/or miscellaneous observations can you note? 5) How 
familiar are you with this comic strip? 6) How long did it take you to read the 
book? This paper focuses on students' responses to Question #3. 

The task consisted of students individually reading the comic strip book 
and writing their responses to the above questions outside class. Afterwards, 
during one class session, approximately an hour and a half, the individual 
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responses were discussed in groups. Each group was asked to teach 
consensus on a cultural description of this comic strip, following the set of 
questions they had worked through individually. They were asked to put their 
group response in writing. 

Grouping, for purposes of this in-class activity, was by self-selection !n 
order to make the task more fun, students were assigned to groups of 4-7, on a 
first-come, first-serve basis, by raising their hands as they agreed with 
particular, casual statements (in the second study, students "numbered off"). 
Class discussion followed the group processing, including inquiry into change 
in their thinking as a result of attempting group consensus, i.e., group 
description of the culture. 

The above is a description of Study 1 , conducted Spring 1 993. Study 2, 
conducted in Summer 1993, is a replication of Study 1, with the addition of 
some training in coding techniques. Basically, the training in Study 2 involved 
repeated viewings of a video of a culture which was unfamiliar to students in the 
class; it involved instructing and coaching them in observation, note-taking, and 
concept development and categorization (Spradley, 1979). 

Study 2 is in two parts, Part I being equivalent to the first study where 
students were asked simply to describe the culture in the first third of the book, 
without reliance on any specific training. In Part II they were asked to apply their 
coding training to interpretation of the culture as depicted in either the second or 
third section of the book. 
Data Analys is 

Students' written responses to Question #3 were categorized and tallied 
according to 1) individual answers written before the group session, 2) group 
answers found by tallying the responses of individual group members, and 
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3) group answers reported as group consensus by the group's recorder. 
Categories of cultural values were generated from students' responses. The 
results from Study 1 are reported in Table 1 . The results from Study 2, in two 
parts, are shown in Tables 2 and 3. 

Results of Study 1 (Without training) 

Findings for the class in Study 1 were: 1) Intra-group differences and 
perceptions were pronounced. Out of 1 9 possible categories, there was no 
group in which everyone selected any one category, although in some cases 
attention clustered in certain categories, 2) Group consensus varied 
considerably from individual reports, 3) Individuals reported more diverse 
answers than group consensus indicated, in spite of the fact that most 
individuals reported no change in atttitudes as a result of the experience, 

4) Group consensus provided responses which were more simplified, and were 
focused on fewer categories than those of individuals, and 5) Unexpectedly, 
inter-group comparisons based on consensus yielded little agreement. 
Results of Study 2 (With training) 

Study 2 produced slightly different categories than Study 1 (See Tables 
1 , 2, and 3). For the class in Study 2 (that received training), more categories 
were identified, implying that training may have increased their awareness and 
ability to discern increased numbers of categories. 

As one can see from Table 2, Study 2, Part I, this class, when given the 
s»me task as those in Study 1 , produced results which were not any more 
consistent than those in Study 1 ; there was still a wide range of differences in 
categories within and between groups. For instance, members of Group 2 
identified 18 out of 29 categories and Group 3 members had 12 of 29 
categories. Looking at the consensus items for Study 1, Part I, i.e., how many 
categories individuals identified compared with categories that the group 
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agreed upon as existing, we found the following percentages of agreement for 
the five groups: 40%, 24%, 54%, 37%, and 35%. The percentages were 
somewhat lower than the amount of agreement between individuals in the 
groups and consensus items for those groups than we found in Study 1 . 

Table 3 summarizes the data that was collected from students in Study 2, 
Part II, where they were asked to apply their training in coding to the 
interpretation of the cartoon culture. As one can see from Table 3, there were 
more categories produced here than in Study 1 or in Part I of Study 2. 
However, the percentages of agreement between the individual responses in 
the group and the consensus items for the group were less than those in Study 
1 or in Part I of Study 2. The percentages for Study 2, Part II were: 27.3%, 
22.2%, 30%, 28%, and 30%. 

Students were asked if they had changed their mind on categories which 
they had generated individually as compared with the group's report of 
consensus items. Interestingly, a large majority of students (77% in Study 2) 
said the process of reaching grouo consensus did not change their own original 
opinion, and in addition, verbal reports from a majority of students indicated that 
the group consensus was reflective of their own opinion. These findings are 
contrary to the individual data indicating that there were many more individual 
differences than the group reported as consensus. This paradox was found in 
Study 1 ; Study 2, Part I; and Study 2, Part ll--with training as well as in the 
absence of training. It is interesting to note that in many cases where students 
reported no change in their views as a result of the group process, they did 
comment that they became aware of additional ideas and categories that they 
had not generated on their own. Although these comments indicated that more 
categories were experienced in the group process, what came out of the group 
were fewer categories-fewer than the individuals generated collectively, and, 
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for the most part, fewer than individuals had generated prior to the group 
experience. 

Discussion/Implications 

In Study 2, in part, we replicated Study 1 , but also looked at the effects of 
training students in coding methodology on interpretation of qualitative data and 
consensus within and between groups. Results of the second study are very 
similar to the first study. Although it appears that training in coding methods 
increased the perceived number of categories, it seemed to have little or no 
effect on agreement within and between groups when compared to the first 
study, where students had not received training. There are a variety of possible 
explanations for this outcome. 

There were different individuals that classified student categories in 
Study 1 and Study 2. This may explain some differences between Study 1 
Study 2. However, this would not account for difference or lack of difference in 
Parts I and II of Study 2; in that case, the person classifying the categories was 
the same. 

There were some overall differences in the makeup of the two classes, 
those in Study 2 being generally older than Study 1 , more frequently in practice 
as teachers, administrators, counselors, or social workers; and, those in Study 1 
tending to be younger, non-practitioner, and in many cases, preparing for 
careers in counseling/psychology. We cannot, at this point, gauge the impact 
that these differences had on outcomes between the two studies, but again, this 
would not have affected comparisons between Parts I and II of Study 2. 

In general, there was a fair amount of naivete in the students asked to do 
the coding, which was reflected in how they did the coding and the consequent 
generation of categories. This naivete was obvious in looking at their attempted 
coding of the cartoon culture, which may be more problematic in these studies 
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than one would suspect to be the case for those who actually do qualitative 
research. 

As one can see from our data, individuals thought they actually received 
more information by discussing in group, yet what always came out of the group 
was a sub-set of all the individual responses. It appears that one does get 
more information by discussing things in groups, but that the aggregated 
consensus in the group is less than the total of all the individual responses, and 
that agreement between groups on the consensus items was not any better 
than agreement between consensus items for individuals in the groups. 
Actually, the areas where the individuals had the highest agreement are not 
reflected in the consensus statement, which would argue against generalizing 
from qualitative research. 

The person who analyzed the student classifications for Study 2 reported 
difficulty and frustration since some of the st» :dents' classifications depended 
upon so many nuances and subtleties. The use of a cartoon culture, especially 
a satirical one, may have presented some unique problems in this regard. For 
instance, some students may have been responding to the culture in a 
concrete way; i.e., they interpreted "sexist" references as reflecting the meaning 
of the culture. Others, realizing it was satirical, appear to have responded to the 
satire; i.e., picking up on very sexist comments, they would say that the cartoon 
is arguing against sexism by making it sound stupid. Others seemed to deal 
with it in terms of a counter-culture and its political implications. So the same 
event, depending on the students' perspectives, may have been coded totally 
differently. One may argue that this is a problem. We are taking the position 
that virtually all qualitative researchers have a perspective, whether aware or 
not, and that these different perspectives are likely to produce different coding 
responses. The fact that we used a caitoon culture may only make these 



ERIC 



11 



10 



discrepancies more obvious. This experience with the cartoon culture might be 
a more accurate simulation of the real problems encountered in me 
interpretation of qualitative data than we originally realized. 

Last, we do not know how much of these results we can attribute to the 
quality or extent of training received, since the time allotted for training was 
minimal-approximately an hour out of each of three class sessions. 
Importance of the Study 

This data support a variety of conjectures. First, if we have individuals 
interpreting culture, we could easily get a perception that is difficult to agree 
upon because it is based on idiosyncratic analysis of data describing cultural 
vaiues. Secondly, recent mainstream thinking of qualitative and quantitative 
researchers suggests that (a) individuals are more likely to give rich, in-depth 
perceptions, (b) however, that individual data is less likely to be generalizable 
to other individuals perceiving this data. Group data may be somewhat more 
generalizable. We therefore might consider using aggregated data with the 
expectation to get data that tend to be more consistent and reliable, but less rich 
with idiosyncratic differences. However, our data suggests that generalizability 
was not increased by the aggregated group consensus data. 

Third, the group produced less in the way of in-depth perceptions, in 
these studies, fewer and more general categories than did individuals. Fourth, 
the results are contrary to recent attempts by some qualitative researchers, such 
as Polkinghorne (1991), to generalize with qualitative research, although they 
say that it is a different type of generalization. 

Last, one may wish to train observers to increase inter-rater reliability; 
however, trained observers might acquire a pre-set and see what they were 
trained to see, or what they were trained to see as important. It is difficult to 
understand how one can be trained to observe independently of their own 
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personal experiences and values. Training people might increase the reliability 
but at the potential cost of validity. Our results show that training increased the 
percentage of agreement among individual raters while it also increased 
variability (an idiosyncratic aspect). 

This simulated anthropological study was done to estimate the 
accuracies and consistencies between individual and group perceptions on a 
relatively "neutral" culture (with fictitious cartoon characters). As indicated 
earlier, we wanted to determine the relative consistency between individuals v. 
groups, and we found that group predictions were not more consistent than 
individual predictions. We initially expected that it would increase consistency 
(replicability) when doing ethnographic research to have more than one 
observer looking at the same data. These studies, however, did not support this 
initial assumption. This research suggests that triangulation in the collection 
and analysis of data may be more problematic than we initially expected. 

In addition to replicating Study 1, Study 2 was an investigation of the 
effects of training students in coding methods to determine the effects of 
consistency on their perception and their willingness to achieve change for 
consensus. We were interested in a) the effects of coding training on 
participants' awareness of their changes in perception, b) a possible increase in 
the likelihood of group consensus, and c) a possible increase in the 
generalizability of group consensus. 

Our research, we believe, supports the need to do further research on 
qualitative methodology for the purpose of improving appropriate interpretation 
of qualitative research. 
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•1 Questjon #3 is: Describe this culture (what are the cultural values?). 
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*1 Question #3 Is: Describe this culture (what are the cultural values?). 
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ST W$ AVAILABLE 



Elicited Categories: 



TABLE— 1 

1. Enjoy Life, have fun 

2 . Anti-Materialistic 

( Ant i -ma ins t ream, ant i - 
defense anti-republican) 

3 ♦ Speak for minorities 
(minority viewpoints 
not just ethnic minority) 

4* Individualistic 

5 . Xenophobic 

6. Emphasis on honesty 

7 . Political correct 

8 . Anti-individualistic 

9 . Anti-personal freedom 

10. Materialistic 

11. Women exploited 

12 . Labeling people 

13 . Men aggressive 

14. Emphasis on friendship 

15. Emphasis on environmental 

16. Emphasis on family 

17. Future time orientation 

18. Importance of group 

19. Political organizations 

TABLE 2 

1. Enjoy life/pleasure 

2. Anti-Establishment, anti- 
ma ins t r earn , ant i - gove rnment 

3. Pro-establishment, anti- 
mainstream, ant i -government 

4. Speak for minorities 
(minority viewpoints, not 
just ethnic minority) 

5. Sexism (Male and female) 

6. Importance of relationships 

7 . Xenophobic /labeling 
8 • Honesty/ justice 

9 . Political correctness 

10. Altruism/personal growth 

11. Gender differences 

12. Effects of the media 

13 . Pro-violence 

14. Anti-violence 

15. Lack of existential factors 
and personal growth 

16 . Superficial/ stereotypes 

17. Lack of work ethic 

18. Lack of value of education 

19. Value technology 

20. Hedonism 



21. Loyalty to others 

22 . Pro-environmental 
23 ♦ Anti-materialistic 
24 • Pro-materialistic 

25. Exploit others 

26. Individual rights 

27 . Lack of trust 

28. Lack of responsibility 
2? Sarcasm/ Satire 

TABLE 3 , 

1. Enjoy life/pleasure 

2. Anti-Establishment, anti- 
mains t r earn , an t i -government 

3 . Speak for minorities 

(minority viewpoints, not 
just ethnic minority) 

4. Sexism (Male and female) 

5. Importance of relationships 

6 . Xenophobic /labeling 

7 . Honesty/ justice 

8. Political correctness 

9 . Altruism/personal growth 

10. Gender differences 

11. Effects of the media 

12 . Pro-violence 

13 . Anti-violence 

14. Lack of existential factors 
and personal growth 

15. Superficial/ stereotypes 

16. Lack of work ethic 

17 . Lack of value of education 

18. Value technology 

19. Loyalty to others 

20 . Pro-environmental 

21 . Anti-materialistic 

22 . Pro-materialistic 

23. Exploit others 

24. Individual rights 

25. Lack of trust' 

26. Lack of responsibility 

27 . Sarcasm/satire 

28. Importance of family 

29 . Politics 

30. Social values 

31. Dating 

32. Argument 

33 . Personal/social change 

34. Children 
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